Detection method for transparent window cleaning device, image processing approach

Recent years, there has been an increase in the number of high-rise buildings, and subsequently, the interest in external wall cleaning methods has similarly increased. While a number of exterior wall cleaning robots are being developed, a method to detect contaminants on the exterior walls is still required. The exteriors of most high-rise buildings today take the form of a window curtain-wall made of translucent glass. Detecting dust on translucent glass is a significant challenge. Here, we have attempted to overcome this challenge using image processing, inspired by the fact that people typically use just the ‘naked eye’ to recognize dust on windows. In this paper, we propose a method that detects dust through simple image processing techniques and estimates its density. This method only uses processing techniques that are not significantly restricted by global brightness and background, making it easily applicable in outdoor conditions. Dust separation was performed using a median filter, and dust density was estimated through a mean shift analysis technique. This dust detection method can perform dust separation and density estimation using only an image of the dust on a translucent window with blurry background.


Overall two-stage structure
Our proposed method is divided into two stages: dust separation and true density estimation. The first stage is to separate sections that look like dust-as a binary image-from images of a translucent glass window. The second stage is to estimate the true dust density of the entire window by using the binary image obtained earlier.
Dust separation. Dust particles are small. Most cameras set the focal point close to the window surface so as to capture these small dust particles. This would focus the camera on the translucent window surface and sharpen the dust particles. The authors concentrated on the fact that the background was blurry and the dust particles were sharp. Therefore, this problem was solved using edge detection or a noise filtering method, as shown in Fig. 1, the details of which are discussed in "Dust separation stage".
True density estimation. Most of the dust adhering to windows in tall buildings has a uniform density. This is based on the fact that the small particles are scattered. However, depending on the color of the background, there are areas where dust particles are more easily visible than others. This means that the camera cannot display all of the dust particles on the screen. Therefore, it is necessary to estimate the true dust density of all dust particles using just some of them. The authors approached this problem by proposing an area representing the true density through clustering, focusing on the fact that the areas where dust particles were clearly captured appear in the form of a cluster. The details are presented in "True density estimation stage".

Test environment
Test bench. The test bench consisted of two parts. First, a fixture that held the camera was located in the middle of a cube made of aluminum profiles. The camera used was a Basler acA1440-220uc USB 3.0 color camera. A reference specimen that simulated dust particles was held in front of the camera. This test bench was placed in various environments, and the experiment was conducted using the acquired images. Figure 2 shows the structure of the test bench.
Reference specimen. If the experiment was conducted using a window with real dust particles, it would not be possible to determine the true dust density. Consequently, we would be unable to analyze the experimental results. Therefore, a substitute with a similar appearance to that of a window with real dust had to be developed. In this study, we used a transparent acrylic plate with laser-engraved dots on it. This was done to control the true dust density of the specimen. To simulate the dust particles, a random scattered pattern was used, simulating the shape of a scatter pattern with a uniform density of actual dust particles.

Dust separation stage
When the camera is focused on the dust particles, the background is out of focus, as shown in Fig. 1a. At this point, the dust particles to be separated are sharp, and the background, which is to be ignored, is blurry. To distinguish between the two, an edge detection method or a noise filter can be used. We experimented using various methods. For edge detection, the Sobel 12 and Prewitt 13 methods were evaluated. Gaussian blur 16 and median filter 14 techniques were used as noise filtering methods. The edge detection method detects sharp dust pixels by www.nature.com/scientificreports/ considering them to be a type of boundary line. The noise filtering method considers dust pixels to be noise and eliminates them, enabling dust pixels to be detected using image differencing techniques. Different methods create different detection images. The edge detection method conducts thresholding on the image on which the method was applied. The noise filtering method conducts thresholding on the difference between the original image and the noise-filtered image. To determine which method to use, we defined two scores. The dust separation score (DSS) is a score for how well the method separates dust particles. The background ignore score (BIS) is a score for how well the method produces a good DSS with only a small degree of background blur. To calculate the score, we synthesized a randomly generated dust mask using various blurry backgrounds, after which, four methods were compared using the corresponding images.

Dust mask & blurry background image set.
For the test, we needed a randomly generated dust mask to overlay the 720p image. For an image with a resolution of 1280 720 pixels, we generated dust particles with random size, blur, and grayscale at uniform random points number of 2304, 9216, 20736, 36864. The size range was [0 1], the blur sigma (Gaussian filter size) range was [0 3] and [0 5], and the grayscale range was [50 255]. An image set was created by combining this dust mask with background images of various background types and focus levels. MATLAB was used for generating and converging test image sets.
The image set to be used for the test was created by synthesizing an image taken by setting only the focus differently at the same position as the dust mask made in this way. Once the true dust pixel information and the sharpness information of the background image were known, the DSS and BIS could be calculated. Figure 3 shows the structure of the image set.
Dust separation score. First, it must be determined how well the method distinguished between the dust particles and the background. The definition of the separation accuracy (SA) of one image, comparing the image detected through our method against the dust mask, with image size of n × m is as follows:   Therefore, the definition of DSS is as follows: Figure 4 shows the t − SA plot and DSS for the four methods. The median filter (filter size 5) showed the best DSS with a value of 0.898. Next, the Sobel and Prewitt filters had a similar DSS of 0.866 and 0.865. Finally the Gaussian filter showed the lowest DSS, 0.86. As the threshold value increases, more detected pixels are filtered, so that the image converges to an image with all pixel values of 0. Since the accuracy is evaluated by the XOR method, even if the part where the original value should be 0 is 0, it is scored. Therefore, it converges to the score for the empty part of the dust mask. It can be confirmed that for each of the four types of methods, as t approaches 255, it converges toward a certain value. Sobel and prewitt filters showed lower values than the minimum DSS received even with a black screen. This means that no matter how well the threshold was set, it was not possible to reduce the misconception that there is dust where there is no dust. However, the gaussian filter and the median filter showed higher DSS than the minimum DSS at a specific threshold value, which means that the noise reduction method has a better overall effect than the boundary detection method. Among them, the median filter showed the best DSS.
Background ignore score. For each edge detection algorithm, a test was conducted to determine how blurry the background needed to be before it could not be detected. This depends on the camera, but the background must focused on to use edge detection. At this stage, the larger the minimum blur level that ignores the background, the larger the minimum distance between the transparent window and the background objects. When detecting the background, the result is different as a part other than the desired dust particles is detected. Therefore, the best method to use is a method that can separate the background and dust pixels with the least blur. In this study, the sharpness of the background was measured using a frequency modulation measuring image quality (FM) method 18 .
We set the sharpness of the background measured through FM on the x-axis. Using the previously introduced t s , an FM-SA plot was created by placing SA(I, M, t s ) as the y-axis. At this stage, the image used is the same background image with different focus levels. Subsequently, the x and y axes are rescaled to a range of [0 1]. For rescaling, the rescale function of MATLAB was used. Consequently, the definition of the BIS is as follows:

Procedure of dust separation stage. Through a series of comparisons, DSS was in the order of Median>
Gaussian> Sobel ≃ Prewitt, and BIS was Gaussian> Median> Sobel ≃ Prewitt. The BIS can be overcome by adjusting the camera's focus and shooting environment, but the DSS is difficult to change. Consequently, a median filter was used.
The image processing procedure of the first stage using a median filter is as follows. The filter receives the original image as an input and outputs a binary image separated by dust.
1. Import original image A. 2. Create A g ray , i.e., a grayscale image of A. 3. Create A m , i.e., a median filtered image of A g ray with filter size 5. 4. Create A diff as abs(A gray A m ). 5. Create A dust as T(A d iff , t s,median = 6).
In Fig. 1a,c represent A g ray , (b) denotes A m , and (d) represents A d ust . The resulting binary image is used in the next stage, i.e., the true density estimation stage.

True density estimation stage
As can be seen in Fig. 1, the result of dust detection is largely aggregated into clusters depending on the background. Therefore, a proposal for an area of maximum density based on the dust binary image from the first stage is required. The detection area proposal proceeds through clustering, and the dust density of the proposed area is returned as the estimated dust density value. There are several types of clustering algorithms, such as the k-means 19,20 , DBScan 21 , and mean shift 17,22 methods. We chose a mean shift approach using a flat kernel. This is because there is no need to predetermine the number of clusters. Moreover, if the initial positions of the centroids are in the form of a grid on the image, this approach can search for the global maximum density.
Mean shift clustering. Clustering using mean shift has been reported previously 17 . Mean shift is an iterative algorithm that moves each centroid point to the average of the data adjacent to the corresponding area. In this study, a mean shift with a flat kernel was used for 2D Boolean data. At this stage, let the data set S to be the circle area radius of and center is m. Then, the mean shift is expressed as follows:

s
: each pixel value of S F(x) : flat kernel : window radius D m : mean shift distance www.nature.com/scientificreports/ This process is repeated for all centroids until convergence-that is, the centroids climb toward a higher density in the density space and stop at the local maximum with the largest density.
Detection area proposal by mean-shift. In this study, mean shift clustering was applied to the density estimation stage by focusing on the property that mean shift clustering converges to a high density. We need the global maximum density, but one centroid will converge to the local maximum. To solve this problem, we uniformly arrange the initial positions of the centroids throughout the image to secure centroids that converge to the global maximum density. After setting the initial position in the form of a grid on the image, mean shift clustering is performed. When clustering is completed, the maximum density is estimated by sorting the density around each centroid in descending order.
The parameters used in this process are as follows.
1. l: grid length. The initial position of the centroid is arranged in the form of a grid, which is the length of one side of the grid. This was arbitrarily set to 40. 2. : window radius. The radius of the window is observed at one centroid. We set the diagonal length of the grid so that there is no empty space at the initial position. = √ 2l 3. D d : dead distance. When the centroid has almost stopped moving, the point is assumed to be a dead point, further calculations are stopped, and the point is assumed to be the convergence point. Centroids moving below D d converted to the dead state.
At this time, the process of the density estimation stage using mean shift is as follows (moreover, this can be checked in Fig. 6).
1. Import A d ust from the first stage. 2. Initialize centroids C in the form of a grid with parameter l. 3. Perform mean shift process with A d ust using , l, D d . Obtain converged C. 4. Obtain the top 3% of the density of each centroids in C.

Mean of 4) is the estimated dust density d.
There are two ways to perform the mean shift process. The first is a method of performing the mean shift of all centroids that are alive for each iteration. This makes all the points converge as a whole, but the calculation of centroids with a very small D m tends to slow the calculation of centroids with a large D m . If calculations cannot be conducted until the end and are stopped prematurely, the convergence progress of each centroid will be different. The second way to perform this method is to use a priority queue. Centroids are placed in the priority queue, and only the centroid with the largest D m is dequeued and processed. In this process, if D m is smaller than D d , it is processed as the corresponding dead centroid. In other cases, it is the alive centroid, and thus the D m updated is enqueued in the priority queue once again. This process continues until the priority queue is empty. With this method, even if the calculation process is stopped in the middle, the convergence progress of each centroid remains similar. The pseudo code for each case can be found in Appendix 1. In this study, the first method that calculating the total mean shift for each iteration was used.

Estimation accuracy test.
To test the estimation accuracy of the second stage, the test bench of Fig. 2 was used. We chose transparent acrylic as the dust mask specimen. The dots were engraved using a laser cutter, and this acrylic dust specimen was used for testing in an actual shooting environment. When the dot density of the acrylic dust specimen with the lowest density was 1, it had a density level (DL) of 1, 2, 3, 4, and 5. Appendix 2 shows each DL demonstration engraved on an acrylic plate. Next, using a test bench, 12 photos were taken with random backgrounds for one dust mask. www.nature.com/scientificreports/ After all dust density estimations were performed for one DL, the results were averaged; this average was defined as the estimated density (ED) of the DL. Consequently, the result was validated, as shown in Fig. 7. As the DL of the acrylic dust specimen increased, the ED also increased at a similar rate. A linear fit was performed using the least squares method, and the coefficient of determination was 0.984. In addition, by processing with a picture of an actual window, it was also confirmed that the amount of dust and the dust estimation result were proportional, as shown in Fig. 8.
Weather conditions were not considered. This is because the exterior wall cleaning robot is not operated in snowy and rainy weather for safety reasons. Especially in rainy weather, there is no need to proceed with cleaning. Sunny and cloudy weather conditions are the difference in whether strong direct sunlight is reflected on the glass window, and if a cover that blocks direct sunlight is used on the robot's camera part, the difference becomes meaningless. Therefore, this experiment was conducted focusing on the brightness condition rather than the weather.

Conclusion & discussion
In this study, a method for estimating the dust density of a translucent glass window was developed using simple image processing methods. Different from previous study 8 , this method has more free brightness condition for implementation in real-world facade cleaning robot. A dust separation stage was performed through a median filter, and dust density estimation was performed through mean shift clustering. Experiments conducted with a test bench showed that the estimated dust density was directly proportional to the true dust density. Moreover, a good coefficient of determinant of 0.984 was obtained as a result of the linear fit. This method is relatively free of global brightness and background influence and can operate under various brightness conditions, such as on the exterior wall of a building. This method can be applied if the camera is focused on the dust on the glass surface, and the background is sufficiently blurred. Moreover, a low-quality camera may be used, as long as a sufficient shutter speed can be obtained so that dust does not itself become blurred when shooting on the move. However, this method does have some limitations. First, if there is an object too close to the window, the object is clearly visible even if the camera focuses on the window surface. In this case, edge detection of the object occurs and greatly disturbs the dust estimation result. Second, if a light source is captured directly by the camera or reflected